A Comment on the ROC Curve and the Area Under it as Performance Measures

The Receiver Operating Characteristic (ROC) curve is a two dimensional measure of classification performance. The area under the ROC curve (AUC) is a scalar measure gauging one facet of performance. In this note, five idealized models are utilized to relate the shape of the ROC curve, and the area under it, to features of the underlying distribution of forecasts. This allows for an interpretation of the former in terms of the latter. The analysis is pedagogical in that many of the findings are already known in more general (and more realistic) settings; however, the simplicity of the models considered here allows for a clear exposition of the relation. For example, although in general there are many reasons for an asymmetric ROC curve, the models considered here clearly illustrate that for symmetric distributions, an asymmetry in the ROC curve can be attributed to unequal widths of the distributions. Also, for bounded forecasts, e.g., probabilistic forecasts, any asymmetry in ROC can be explained in terms in terms of a simple combination of the means and widths of the distributions. Furthermore, it is shown that AUC discriminates well between “good” and “bad” models, but not between “good” models.